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DETAILED ACTION 
Election/Restrictions 

1 . Claims 1 2-28 and 39-54 are witlidrawn from furtlier consideration pursuant to 37 
CFR 1.142(b) as being drawn to a nonelected invention, tliere being no allowable 
generic or linking claim. Election was made without traverse in the reply filed on 13 
May 2008. 

Priority 

2. Receipt is acknowledged of papers submitted under 35 U.S.C. 1 19(a)-(d), which 
papers have been placed of record in the file. 

Claim Rejections - 35 USC § 101 

3. 35 U.S.C. 101 reads as follows: 

Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, 
or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and 
requirements of this title. 

4. Claims 57 and 58 are rejected under 35 U.S.C. 101 because the claimed 
invention is directed to non-statutory subject matter. Claims 57 and 58 fail to fall within 
a statutory category of invention. They are directed to the program itself. They are not 
a process occurring as a result of executing the program, a machine programmed to 
operate in accordance with the program, nor a manufacture structurally and functionally 
interconnected with the program in a manner which enables the program to act as a 
computer component and realize its functionality. They are also clearly not directed to a 
composition of matter. Therefore, they are non-statutory under 35 USC 101 . 
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Because the full scope of the claims as properly read in light of the disclosure 
encompasses non-statutory subject matter, each claim as a whole is non-statutory. 

Any amendment to the claims should be commensurate with the corresponding 
disclosure. 

Claim Rejections - 35 USC § 102 

5. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 1 02 that 
form the basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(b) the Invention was patented or described in a printed publication in this or a foreign country or in public 
use or on sale in this country, more than one year prior to the date of application for patent in the United 
States. 

6. Claims 1 , 29-31 , and 55-58 are rejected under 35 U.S.C. 1 02(b) as being 
anticipated by Lu (US 5,550,928). 

Regarding claim 1, Lu teaches an audience state estimation system (Fig. 

1, 3) comprising: 

imaging device for imaging an audience and generating a video signal 
relative to the audience thus imaged (Col. 8, line 54-Col. 9, line 31); 

movement amount detection device for detecting a movement amount of 
said audience based on said video signal, i.e. tracking a person (Col. 10, lines 6- 
25); and 

estimation device for estimating an audience state based on said 
movement amount, i.e. determining who is in the area and what they are doing 
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based on the face and body recognition along witli tlie motion tracl<ing (Col. 9, 
line 60-Col. 1 0, line 49; Col. 11 , line 47-Col. 1 2, line 59; Col. 1 4, lines 37-65). 



Regarding claims 29, 30, 55-58, claims are analyzed with respect to claim 
1 . These are Markush claims that include estimating the audience state based 
on audio taken from the audience, data of which was assigned to the non-elected 
Group II of the Restriction/Election Requirement, and likewise all limitations 
dealing with audio will not be examined. 



Regarding claim 31 , claim is analyzed with respect to claim 1 . 



Claim Rejections - 35 USC § 103 

7. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or deschbed as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the phor art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

8. Claims 2, 4, 5, and 32-34 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Lu in view of Tow (US 7,266,771 ). 

Regarding claim 2, Lu teaches all elements of claim 1 . 

Lu further teaches the movement amount detection device determines 



movement information of the imaged audience based on said video signal, i.e. 
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images are subtracted one from anotlier to determine if motion has occurred 
(Col. 10, lines 6-25). 

Tow teaches using MPEG differential frames that store motion information 
in the form of motion vectors obtained by determining the difference between 
adjacent frames (Col. 6, lines 47-60; Col. 8, lines 14-65); and 

wherein an average movement amount showing an average of 
magnitudes of the movement vectors is set as the movement amount (Col. 9, line 
55-Col. 10, line 33). 

Therefore, It would have been obvious to one of ordinary skill in the art at 
the time the invention was made to modify Lu to include determining motion 
vectors; and wherein an average movement amount showing an average of 
magnitudes of the movement vectors is set as the movement amount of said 
audience, as taught by Tow, for the purpose of identifying a type of motion 
energy at a particular place in a video and presenting it to a viewer in a 
graphically intuitive manner (Tow-Col. 3, lines 29-35). 

Regarding claim 4, Lu teaches all elements of claim 1 . 

Lu further teaches using a sequence of video images to track and locate a 
target over a time interval (Col. 12, lines 13-24). 

Lu does not clearly teach the movement amount detection device 
determines movement vectors of the imaged audience based on said video 
signal and calculates an average movement amount showing an average of 
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magnitudes of the movement vectors, and wherein a time macro movement 
amount is set as the movement. 

Tow teaches using MPEG differential frames that store motion information 
in the form of motion vectors obtained by determining the difference between 
adjacent frames (Col. 6, lines 47-60; Col. 8, lines 14-65); and 

calculating an average movement amount showing an average of 
magnitudes of the movement vectors (Col. 9, line 55-Col. 10, line 33), and 

wherein a time macro movement amount is set as the movement amount 
of said audience, said time macro movement amount being an average of the 
average movement amounts in a time direction thereof, i.e. the motion vectors 
have a magnitude and direction over the time period of a frame or several frames 
(Col. 9, line 55-Col. 10, line 33). 

Therefore, it would have been obvious to one of ordinary skill in the art at 
the time the invention was made to modify Lu to include the movement amount 
detection device determines movement vectors of the imaged audience based on 
said video signal and calculates an average movement amount showing an 
average of magnitudes of the movement vectors, and wherein a time macro 
movement amount is set as the movement, as taught by Tow, for the purpose of 
identifying a type of motion energy at a particular place in a video and presenting 
it to a viewer in a graphically intuitive manner (Tow-Col. 3, lines 29-35). 



Regarding claim 5, Lu teaches all elements of claim 1 . 
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Lu does not clearly teach when said movement amount is larger than a 
predetermined level, said estimation device estimates said audience state to be 
in any one of states of beating time with the hands and of clapping. 

Tow teaches when said movement amount is larger than a predetermined 
level, said estimation device estimates said audience state to be In any one of 
states of beating time with the hands and of clapping, i.e. using a motion 
information template that corresponds to clapping (Col. 10, line 46-Col. 11, line 
3). 

Therefore, it would have been obvious to one of ordinary skill in the art at 
the time the invention was made to modify Lu to include when said movement 
amount is larger than a predetermined level, said estimation device estimates 
said audience state to be in any one of states of beating time with the hands and 
of clapping, as taught by Tow, for the purpose of identifying a type of motion 
energy at a particular place in a video and presenting it to a viewer in a 
graphically intuitive manner (Tow-Col. 3, lines 29-35). 

Regarding claim 32, claim is analyzed with respect to claim 2. 

Regarding claim 33, claim is analyzed with respect to claim 4. 



Regarding claim 34, claim is analyzed with respect to claim 5. 
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9. Claim 3 is rejected under 35 U.S.C. 103(a) as being unpatentable over Lu in view 
of Tow and further in view of Toyama (US 6,792,1 35). 

Regarding claim 3, Lu in view of Tow teaches all elements of claims 1 and 

2. 

Lu in view of Tow teaches when an area identified based on color 
information is divided into blocks, the movement vector is determined for each of 
the blocks (Tow-Fig. 2, el. 201, 203, 205, 207, 209; Col. 8, lines 14-65). 

Lu in view of Tow does not clearly teach identifying an area based on 
color information. 

Toyama teaches identifying an area based on color information (Col. 7, 
lines 1-24). 

Therefore, it would have been obvious to one of ordinary skill in the art at 
the time the invention was made to modify Lu in view of Tow to include 
identifying an area based on color information, as taught by Toyama, for the 
purpose of identifying and tracking an object in a video. 



10. Claims 6-10 and 35-37 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Lu in view of Tow and further in view of Tagawa (US 7,373,209). 

Regarding claim 6, Lu teaches an audience state estimation system (Fig. 
1, 3) comprising: 

imaging device for imaging an audience and generating a video signal 
relative to the audience thus imaged (Col. 8, line 54-Col. 9, line 31). 
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Lu does not clearly teach a movement periodicity detection device for 
detecting movement periodicity of said audience based on said video signal; and 
estimation device for estimating an audience state based on the movement 
periodicity of said audience. 

Tow teaches using MPEG differential frames that store motion information 
in the form of motion vectors obtained by determining the difference between 
adjacent frames (Col. 6, lines 47-60; Col. 8, lines 14-65); and 

the motion vectors have a magnitude and direction over the time period of 
a frame or several frames (Col. 9, line 55-Col. 10, line 33); and 

estimation device for estimating an audience state based on the 
movement periodicity of said audience, i.e. using a motion information template 
that corresponds to clapping (Col. 10, line 46-Col. 1 1 , line 3). 

Therefore, it would have been obvious to one of ordinary skill in the art at 
the time the invention was made to modify Lu to include determining motion 
vectors; and wherein an average movement amount showing an average of 
magnitudes of the movement vectors is set as the movement amount of said 
audience, as taught by Tow, for the purpose of identifying a type of motion 
energy at a particular place in a video and presenting it to a viewer in a 
graphically intuitive manner (Tow-Col. 3, lines 29-35). 

Lu in view of Tow does not clearly teach a movement periodicity detection 
device for detecting movement periodicity of said audience based on said video 
signal. 
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Tagawa teaches detecting periodicity based on an audio signal, i.e. 
identifying a periodicity of a rliythm or beat in music based on the peaks of an 
auto-correlation function of the audio (Col. 13, lines 5-49). 

Therefore, it would have been obvious to one of ordinary skill in the art at 
the time the invention was made to modify Lu in view of Tow to include a 
movement periodicity detection device for detecting movement periodicity of said 
audience based on said video signal, using the technique taught by Tagawa in 
combination with the motion vector system taught by Lu in view of Tow, although 
in different fields of endeavor would provide a predictable variation to the motion 
vector system and for the purpose of specifically identifying a type of motion for 
use in a video retrieval system. 

Regarding claim 7, Lu in view of Tow in view of Tagawa teaches the 
movement periodicity detection device determines movement vectors of the 
imaged audience based on said video signal (Tow-Col. 6, lines 47-60; Col. 8, 
lines 14-65; Tagawa-Col. 13, lines 5-49), 

calculates an average movement amount showing an average of 
magnitudes of the movement vectors (Tow-Col. 9, line 55-Col. 10, line 33), and 

detects an autocorrelation maximum position of the average movement 
amount (Tow-Col. 10, line 57-Col. 11, line 3; Tagawa-Col. 13, lines 5-49), and 

wherein variance of the autocorrelation maximum position is set as said 
movement periodicity (Tagawa-Col. 13, lines 5-49). 
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Regarding claim 8, Lu in view of Tow in view of Tagawa teaclies tine 
variance is calculated using a signal in a frame range, said frame range being 
decided on the basis of the periodicity of said audience state to be estimated 
(Tow-Col. 9, lines 37-55; Col. 10, lines 45-67; Tagawa-Col. 13, lines 5-49). 

Regarding claim 9, Lu in view of Tow in view of Tagawa teaches the 
movement periodicity detection device determines movement vectors of the 
imaged audience based on said video signal (Tow-Col. 6, lines 47-60; Col. 8, 
lines 14-65), and 

calculates an average movement amount showing an average of 
magnitudes of the movement vectors (Tow-Col. 9, line 55-Col. 10, line 33), and 

wherein a ratio of low-frequency component in the average movement 
amount is set as said movement periodicity (Lu-Col. 11, line 48-Col. 12, line 13; 
Tow-Col. 9, line 55-Col. 10, line 33; Tagawa-Col. 13, lines 5-49). Note: The 
image is low-pass filtered and the movement periodicity is determined to be the 
ratio of average magnitudes of the motion vectors distributed over time, such as 
for clapping. 

Regarding claim 10, Lu in view of Tow in view of Tagawa teaches a 
frequency range of the low-frequency component is decided according to the 
periodicity of the said average movement amount transformed to a frequency 
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region to be detected, i.e. identifying tlie rate and rliytlinn of clapping (Lu-Col. 1 1 , 
line 48-Col. 12, line 13; Tow-Col. 9, line 55-Col. 10, line 33; Tagawa-Col. 13, 
lines 5-49). 

Regarding claim 35, claim is analyzed with respect to claim 6. 

Regarding claim 36, claim is analyzed with respect to claim 7. 

Regarding claim 37, claim is analyzed with respect to claim 9. 

1 1 . Claims 1 1 and 38 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Lu in view of Tow in view of Tagawa and further in view of Stevens (WO 
91/03912). 

Regarding claim 1 1 , Lu in view of Tow in view of Tagawa teaches all 
elements of claim 6. 

Lu in view of Tow in view of Tagawa teaches determining the periodicity, 
rate, and rhythm of a set of motion vectors (Tow-Col. 10, line 46-Col. 1 1 , line 3; 
Tagawa-Col. 13, lines 5-49). 

Lu in view of Tow in view of Tagawa does not clearly teach the estimation 
device estimates said audience state to be in a state of beating time with the 
hands when said movement periodicity is larger than a predetermined level, and 
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estimates said audience state to be in a state of clapping when said movement 
periodicity is not larger than said predetermined level. 

Stevens teaches estimating a person to be in a state of beating time with 
the hands when said movement periodicity is larger than a predetermined level, 
and estimates a person to be in a state of clapping when said movement 
periodicity is not larger than said predetermined level, i.e. when a first loud or 
sharp sound is received a counter is started; when the counter reaches a 
predetermined level without the system receiving another loud or sharp sound, 
the system determines that the periodicity of the first and a future second sound 
would be too large to be clapping; when a second loud or sharp sound is 
received before the counter reaches the predetermined level, the system 
determines that the periodicity of the two sounds is within a sufficient range to be 
clapping (Abstract). 

Therefore, it would have been obvious to one of ordinary sl<ill in the art at 
the time the invention was made to modify Lu in view of Tow in view of Tagawa 
to include the estimation device estimates said audience state to be in a state of 
beating time with the hands when said movement periodicity is larger than a 
predetermined level, and estimates said audience state to be in a state of 
clapping when said movement periodicity is not larger than said predetermined 
level, using the technique taught by Stevens in combination with the motion 
vector system taught by Lu in view of Tow in view of Tagawa, although in 
different fields of endeavor would provide a predictable variation to the motion 
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vector system and for the purpose of specifically identifying a type of motion for 
use in a video retrieval system. 

Regarding claim 38, claim is analyzed with respect to claim 1 1 . 

Conclusion 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to JEREMY DUFFIELD whose telephone number is 
(571 )270-1643. The examiner can normally be reached on Mon.-Thurs. 8:00 A.M.-5:30 
P.M. EST. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Scott Beliveau can be reached on (571 ) 272-7343. The fax phone number 
for the organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EEC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 
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4 August 2008 
JSD 

/Scott Beliveau/ 

Supervisory Patent Examiner, Art Unit 2623 



